Trend detection via machine learning models

ABSTRACT

Machine learning-based approaches are used to detect trends of behavior and anomalous events from customer support messages between customers and customer support agents or other appropriate resources in an electronic environment. For example, for a plurality of time periods, a prediction model can be trained. The prediction models can be trained on messages that correspond to each prediction models&#39; period of time. The prediction models can process messages to determine a score (e.g., a representative confidence score) for the time period a prediction model is associated with. For a selected time period, a model (e.g., a trend detection model) can be applied to the scores for time periods before the selected time period to determine whether the score for the selected time period is associated with an anomalous event. Thereafter, an alert can be presented with, for example, the messages that triggered the alert, among other such information.

BACKGROUND

As the number of services and products offered to customers increases,there is a corresponding increase in the need to efficiently interactwith those customers. For example, customers may need to obtaininformation about services and/or products offered by a company, mayhave a question about billing, or may need technical support from thecompany. In any such situation, customers may interact with thesecompanies in a variety of different ways. For example, customers mayinteract with a company via a company website, application, telephonecall, customer support provider, etc., to submit customer supportrequests and obtain information via an interactive interface or customersupport agent.

However, due to the large amount of customer support requests handled bycustomer support agents, it may be expensive in terms of time andresources to identify customer issues in real-time. In many situations,it may be even more challenging to sift through past customer supportmessages to identify trends in customer support messages and prepare forsimilar customer issues reflected in those messages. In conventionalapproaches, a customer support agent may escalate reoccurring issues.Customer support teams may also hold daily, weekly, etc., meetings toreview trends of behavior and develop processes to prepare for similarissues in the future. However, not all customer support agents mayescalate requests due to workload, time, and so on. As such, certainissues may be missed, and procedures may not be developed to handlesimilar issues.

Further, customer support service providers may not have the resourcesto review past messages to identify new and similar issues. Thus, whilesome issues may be resolved, customer support service providers lack thelevel of resolution into current and past customer support messages toquickly respond to current issues, as well as find interpretable trendsof behavior based on past messages, costing the company time and money.Accordingly, it is desirable to provide improved techniques for managingcustomer support messages.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments in accordance with the present disclosure will bedescribed with reference to the drawings, in which:

FIGS. 1A, 1B, and 1C illustrate example approaches to receiving customersupport requests in accordance with various embodiments;

FIG. 2 illustrates an example environment in which aspects of thevarious embodiments can be implemented;

FIG. 3 illustrates an example approach to detecting anomalous event andtrends of behavior in accordance with various embodiments;

FIGS. 4A, 4B, and 4C illustrate examples of various visualizations thatcan be presented in accordance with an embodiment;

FIG. 5 illustrates an example of processing customer support messages inaccordance with various embodiments;

FIG. 6 illustrates an example classification pipeline that can beutilized in accordance with various embodiments;

FIG. 7 illustrates an example process for determining training data thatcan be utilized in accordance with various embodiments;

FIG. 8 illustrates an example process for training a model that can beutilized in accordance with various embodiments;

FIG. 9 illustrates an example environment in which aspects of thevarious embodiments can be implemented;

FIG. 10 illustrates an example process for detecting anomalous events inaccordance with various embodiments;

FIG. 11 illustrates an example process for determining scores that canbe utilized in accordance with various embodiments;

FIG. 12 illustrates an example process for determining scores that canbe utilized alternate embodiments; and

FIG. 13 illustrates an example configuration of components of a device.

DETAILED DESCRIPTION

Systems and methods in accordance with various embodiments of thepresent disclosure may overcome one or more of the aforementioned andother deficiencies experienced in conventional approaches to managingcustomer behavior in an electronic environment. In particular, variousembodiments describe machine learning-based approaches for detectingtrends of behavior and anomalous events from customer support messagesbetween customers and customer support agents or other appropriateresources in an electronic environment.

For example, in accordance with various embodiments, for each one of aplurality of time periods, a model (e.g., a prediction model such as alogistic regression model or another machine learning-based model) canbe trained. The prediction models can be trained on messages thatcorrespond to each prediction models' period of time. The messages canbe obtained from, for example, company records, customer support logs, astream of customer support messages, computer/server communication logs,computer-readable data sent between one or more computers (e.g., messagelogs, instructions, code, etc.), and the like. The messages for eachtime period can be separated into sets of messages. For example, a firstset of messages, a second set of messages, and a holdout set ofmessages, where the first set of messages includes messages receivedbefore a time period, the second set of messages includes messagesreceived during the time period, and the holdout set of messagesincludes messages received during the time period and disjoint from thesecond set of messages.

The prediction models can process messages to determine a score (e.g., arepresentative confidence score) for each time period a prediction modelis associated with. For a selected time period, a model (e.g., a trenddetection model) can be applied to the scores for time periods beforethe selected time period to determine whether the score for the selectedtime period is associated with an anomalous event. Thereafter, an alertcan be presented with, for example, the messages that triggered thealert, business metrics such as a volume of messages causing the alertor a cost for customer support agents to respond to the volume ofmessages, a graphical representation of textual data, an organizeddisplay of textual data, a word cloud, among other such information.

Instructions for causing a computer system to detect trends of behaviorand anomalous events from customer support messages between customersand customer support agents or other appropriate resources in accordancewith the present disclosure may be embodied on a computer-readablemedium. For example, in accordance with an embodiment, a backend systemmay train prediction models for a plurality of time periods. Theprediction models can be trained on customer support messages. Thebackend system can utilize models trained for different periods of timeand other components to determine a score (e.g., a representativeconfidence score) for each time period. For a selected time period, amodel (e.g., a trend detection model) can be applied to the scores forperiods of time before the selected time period to determine whether thescore for the selected time period is associated with an anomalousevent. Thereafter, a visualization component, notification component, orother appropriate components of the backend system can generate an alertthat can be presented with, for example, messages that triggered thealert, business metrics such as a volume of messages causing the alertor a cost for customer support agents to respond to the volume ofmessages, a graphical representation of textual data, an organizeddisplay of textual data, a word cloud, among other such information.

It should be noted that although the techniques described herein may beused for a wide variety of messages, for clarity of presentation,messages described herein are received from customers desiring customersupport for a product, a service, or other offerings from a companyand/or appropriate entity. The techniques described herein, however, arenot limited to customer support messages, and the messages may be fromusers who are not customers and detecting trends of behavior andanomalous events from customer support messages may be performed by athird-party on behalf of another company.

Embodiments provide a variety of advantages. For example, in accordancewith various embodiments, machine learning-based approaches toautomatically detect trends of behavior and anomalous events fromcustomer support messages between customers and customer support agentsor other appropriate resources in an electronic environment can beutilized by any system that attempts to optimize resource usageincluding computing system resources such as feedback systems, resourceallocation systems, response systems, and the like. These systems may beutilized by, for example, customer service platforms, airline bookingplatforms, accommodation booking platforms, financial platforms, medicalplatforms, electronic marketplaces, online bidding platforms, and otherenvironments where customer service may be needed. In accordance withvarious embodiments, by providing a system that automatically detectstrends of behavior and anomalous events from customer support messages,the system can more efficiently utilize resource capacity. For example,approaches described herein can improve the overall performance ofresponding to requests of customers seeking customer support from acompany through near-instantaneous responses to such requests. Forexample, an alarm can immediately be triggered when something wrong isdetected, and the event that triggered the alarm can be addressed.Additionally, new and similar behaviors can be detected and appropriateaddressed. The process is improved by using machine learning-basedtechniques to optimize resource utilization of various resources. Incertain embodiments, one or more trained models can be updated over atleast one period of time using computer-readable information based onone or more performance metrics to ensure optimal usage of variousresources. Further, approaches described herein advantageously providefor improved responses to potential customer issues. This can reduceidle resources, or other such inefficiencies of the resources, which canmaximize resource usage and reduce cost.

Various other functions and advantages are described and suggested belowas may be provided in accordance with the various embodiments.

FIGS. 1A, 1B, and 1C illustrate example approaches to receiving customersupport messages that can be utilized in accordance with variousembodiments. In accordance with various embodiments, the messages can beobtained from customer support sessions between a customer and customersupport agent, and can generally describe issues that customers areexperiencing. The messages can be from a text exchange between acustomer and customer support agent, such as through instant messaging,email, and the like; a conversation, such as one over a phone, internet,etc., or a combination thereof. For example, as shown in example 100 ofFIG. 1A, customer 102 is using computing device 104 to communicate witha customer support agent or an automated customer support service. Inthis example, the customer is inquiring about an issue. In particular,the customer is sending text-based message 106 to inquire whether there“is something wrong with the pay-per-view.”

In example 120 of FIG. 1B, customer 102 is talking into a headset 122 tosend a voice-based message 124 to a customer support agent. Although aheadset is shown, it should be understood that any electronic devicecapable of receiving and/or processing audio data can be used inaccordance with various embodiments discussed herein, where the devicescan include, for example, mobile phones, tablets, microphones, notebookcomputers, personal data assistants, among others. The headset, in thisexample, can include at least one microphone and/or speaker. The headsetcan further include one or more internal processing components (notshown), such as an analog-to-digital (A/D) converter, an audioprocessing engine, an audio classification engine, a time domainconversion module, etc. Further components and software both related andnot related to voice capture, according to various embodiments, may alsobe included in the headset or otherwise in communication with theheadset.

In example 140 of FIG. 1C, an audio communications device 142, issending a voice-based message 124 to a customer support agent. In anembodiment, audio communication device 142 can include, for example,portable computing devices, notebook computers, ultra books, tabletcomputers, mobile phones, personal data assistants, video gamingconsoles, televisions, set top boxes, smart televisions, portable mediaplayers, and wearable computers (e.g., smart watches, smart glasses,bracelets, etc.), display screens, displayless devices, other types ofdisplay-based devices, smart furniture, smart household devices, smartvehicles, smart transportation devices, and/or smart accessories, or anydevice having a microphone, speaker, or other component(s) or softwareconfigured to analyze, generate, and communicate audio data in thefrequency range in which humans communicate or any detectable frequencyrange.

Text from the customer support messages can be obtained using textextraction techniques, speech recognition techniques, and the like. Forexample, in example 100 of FIG. 1A, customer 102 is typing a message tothe customer support agent. A text extraction technique can be used toextract the text from the message. In examples 120 and 140, voice-basedmessage 124 can be received at a speech-to-text component 126 configuredto use one or more speech recognition techniques to convert speech totext. In an example, an A/D converter of the audio analyzing componentcan sample the audio data to convert the audio data into digital audiodata. At least one speech recognition technique can be used to convertthe digital audio data to text. It should be noted that any one of anumber of text extraction techniques and speech recognition techniquescan be utilized in accordance with embodiments described herein.

Once the messages are obtained and the text determined, the text fromthe customer support messages can be analyzed to detect anomalous eventsand trends of behavior. In certain embodiments, an alert can betriggered to alert an appropriate entity of an anomalous event. Invarious embodiments, business metrics such as the volume of messagescausing the alert, a cost for customer support agents to respond to thealert or respond to the volume of messages, can be provided.

FIG. 2 illustrates an example environment 200 in which aspects of thevarious embodiments can be implemented. In this example, a user canutilize a client device 202 to communicate across at least one network204 with a resource provider environment 206. The client device 202 caninclude any appropriate electronic device operable to send and receiverequests, messages, or other such information over an appropriatenetwork and convey information back to a user of the device. Examples ofsuch client devices 202 include personal computers, tablet computers,smartphones, notebook computers, and the like. The user can include aperson authorized to manage the aspects of the resource providerenvironment.

The resource provider environment 206 can provide customer supportservices for companies for various services. These services can include,for example, payment services, technical support services, humanresource services, among other such services, products, and/orofferings. In certain embodiments, resource provider of environment 206can be an intermediary between a customer of a company and the company.The provider can, for example, assist a company by providing customersupport for the company. This can include, for example, identifying andalerting the company of problematic issues raised by customers,detecting trends of behavior/issues from customer support sessions,providing business insights based on the problematic issues and trends,automatically responding to customer support requests, connecting acustomer with a customer service representative (CSR), customer supportagent, etc., selecting a CSR to process a customer support request,establishing customer support sessions between customers and CSRs,providing resources to CSRs to process customer support requests, etc.In various embodiments, the functions of a CSR can be performed inhardware and software, such as by using a trained model, a person, orcombination thereof.

The network(s) 204 can include any appropriate network, including anintranet, the Internet, a cellular network, a local area network (LAN),or any other such network or combination, and communication over thenetwork can be enabled via wired and/or wireless connections.

The resource provider environment 206 can include any appropriatecomponents for receiving customer support requests and/or messages andreturning information or performing actions in response to thoserequests. It should be noted that although the techniques describedherein may be used for a wide variety of users and requests, for clarityof presentation, examples of companies providing a response to acustomer support request of a customer will be used. The techniquesdescribed herein, however, are not limited to customers and companies,and responses may be provided to requests from users who are notcustomers, and responses may be from any entity or person.

The resource provider environment 206 might include Web servers and/orapplication servers for receiving and processing requests thenestablishing customer support sessions and/or assigning customer supportrequests. While this example is discussed with respect to the internet,web services, and internet-based technology, it should be understoodthat aspects of the various embodiments can be used with any appropriateservices available or offered over a network in an electronicenvironment.

In various embodiments, resource provider environment 206 may includevarious types of resources 214 that can be used to facilitate customersupport sessions between customers and customer support agents.

The resources can include, for example, a communications session queueoperable to maintain a queue of customer service requests, acommunications session server operable to facilitate establishingcustomer support sessions and/or assigning customer support requests, amanagement component operable to obtain customer support messages,application servers operable to process instructions provided by a useror database servers operable to process data stored in one or more datastores 216 in response to a user request. The resources may be hosted onmultiple server computers and/or distributed across multiple systems.Additionally, the components may be implemented using any number ofdifferent computers and/or systems. Thus, the components may beseparated into multiple services and/or over multiple different systemsto perform the functionality described herein. In some embodiments, atleast a portion of the resources can be “virtual” resources supported bythese and/or components.

In at least some embodiments, an application executing on the clientdevice 202 that needs to access resources of the provider environment206, for example, to initiate an instance of a trend detection service,can submit a request that is received to interface layer 208 of theprovider environment 206. The interface layer 208 can includeapplication programming interfaces (APIs) or other exposed interfacesenabling a user to submit requests, such as Web service requests, to theprovider environment 206. Interface layer 208 in this example can alsoinclude other components as well, such as at least one Web server,routing components, load balancers, and the like.

When a request to access a resource is received at the interface layer208 in some embodiments, information for the request can be directed toresource manager 210 or other such systems, service, or componentconfigured to manage user accounts and information, resourceprovisioning and usage, and other such aspects. Resource manager 210 canperform tasks such as communicating the request to a managementcomponent or other control component which can be used to manage one ormore instances of a trend detection service as well as other informationfor host machines, servers, or other such computing devices or assets ina network environment, authenticate an identity of the user submittingthe request, as well as to determine whether that user has an existingaccount with the resource provider, where the account data may be storedin at least one data store 212 in the resource provider environment 206.

For example, the request can be to instantiate a trend detection serviceon host machine 220. The trend detection service can utilize trenddetection module 224, notification module 228, visualization module 226,training module 230, and communication manager 232, to detect anomalousevents from customer support messages, identify trends ofbehavior/events from the customer support messages, generate visualsummaries of the events, provide business insights, etc. It should benoted that although host machine 220 is shown outside the providerenvironment, in accordance with various embodiments, one or more modulesof the trend detection service can be included in provider environment206, while in other embodiments, some of the modules may be included inthe provider environment. It should be further noted that host machine220 can include or at least be in communication with other components,for example, a customer support session manager, etc.

FIG. 3 illustrates an example approach 300 to detecting anomalous eventsand trends of behavior in accordance with various embodiments. It shouldbe understood that reference numbers are carried over between figuresfor similar components for purposes of simplicity of explanation, butsuch usage should not be construed as a limitation on the variousembodiments unless otherwise stated. In this example, graph 302illustrates a graphical representation of trends of behavior determinedfrom messages between customers and customer support agents. The x-axisrepresents time and the y-axis represents a level of confidence. Asshown, the x-axis illustrates a time range of one year, and a timeperiod of a week every four weeks. It should be noted that the timerange, time period, and time period interval can be other amounts oftime, and the times used are for example purposes. For example, the timeperiod and/or time period interval can be amounts measured in, e.g.,real-time, seconds, minutes, hours, daily, etc. The y-axis can berepresented as a number, e.g., a number between zero and one, a numberbetween zero and 100, a percentage, etc. Each number can correspond to aconfidence score that represents the likelihood that for a particulartime interval an event occurred. Dots 304, 306, and 308 represent spikesor trends of behavior for a particular time period.

As part of the process for detecting anomalous events and trends ofbehavior, a plurality of messages 314 is obtained. The messages can beobtained from, for example, company records, customer support logs, astream of customer support messages, computer/server communication logs,computer-readable data sent between one or more computers (e.g., messagelogs, instructions, code, etc.), or otherwise obtained. The messages canbe separated into sets. In various embodiments, the messages can beseparated into sets based on the size of a period of time. For example,a first set of messages, a second set of messages, and a holdout set ofmessages, where the first set of messages includes messages receivedbefore a time period, the second set of messages includes messagesreceived during the time period, and the holdout set of messagesincludes messages received during the time period and disjoint from thesecond set of messages. In various embodiments, the period of time canbe, for example, predetermined, user-specified, etc.

Prediction model 316 (e.g., logistic regression over unigrams orbigrams, neural network, etc.) is trained on the first set of messagesand the second set of messages to predict whether a message is receivedbefore time period 318 or during time period 318. For example,prediction model 316 can be used to evaluate each message in the holdoutset of messages to obtain a set of confidence scores corresponding tothe holdout set of messages, where a confidence score can quantify thelikelihood a message is received before or during the time period. Inaccordance with an embodiment, this can include, for example, processingthe text of the messages utilizing Vowpal Wabbit or other machine-basedlearning algorithm, and using a hash function on the text to build abag-of-words (BOW) representation. In this example, each word can beprocessed using a hash function to generate feature vectors for themodel, where the resulting feature vectors are bag-or-wordsrepresentations. Other BOW representations can be used as well,including, for example, TF-IDF weights and/or n-grams etc. The hashedBOW feature vector can then be used to train the model, which can beused in accordance with embodiments described herein to obtain the setof confidence scores.

The set of confidence scores can be processed to determinerepresentative confidence score 320 for time period 318 based on the setof confidence scores. For example, the representative confidence scorefor the time period can be the confidence score corresponding to aconfidence score threshold. In an example, the representative confidencescore can be the confidence score corresponding to the 95th percentileconfidence score. In certain embodiments, the representative confidencescore may be an average or the median of representative confidencescores. For example, a plurality of messages associated with a period oftime can be split into a plurality of groups of first sets of messages,second sets of messages, and holdout sets of messages. A representativeconfidence score can be determined for each holdout set of messages ofthe plurality of groups to obtain a set of representative confidencescores, and the representative confidence score for the period of timecan correspond to one of the average or median of the scores. In yetanother example, representative confidence scores from the set ofrepresentative confidence scores that meet a threshold representativeconfidence score can be selected, and the selected representativeconfidence scores can be averaged. The average representative confidencescore of the selected scores can correspond to the representativeconfidence score for the period of time.

Set of representative confidence scores 322 from a plurality ofprediction models trained over a plurality of previous time periods canbe obtained. Trend detection model 324 can be applied to set ofrepresentative confidence scores 322 to determine whether representativeconfidence score associated with time period is associated with ananomalous event. For example, using the set of representative confidencescores 322, a model can generate data to fit the distribution of thescores. In an embodiment, the model can be a Gaussian mixture model thatcan generate a set of points which approximately fit the set ofrepresentative confidence scores. In the situation a representativeconfidence score does not fit the set of points by a threshold amount,or the model otherwise detects an outlier, or other condition issatisfied, an anomalous event may be detected. In any situation, eventinformation 326 for the period of time can be stored for use in one ormore other purposes. For example, in the situation an anomalous event isdetected, an alert for the anomalous event can be generated andpresented to an appropriate entity. It should be noted that variousother models can be applied as well within the scope of the variousembodiments, such as, e.g., a change point detection model, RANSAC, orother outlier detection model.

FIGS. 4A, 4B, and 4C illustrate various visualizations that can that bepresented in accordance with various embodiments. As described, messagesbetween customers and customer support agents can be analyzed toidentify and alert a company of problematic issues raised by customers,detect trends of behavior/issues from customer support messages, providebusiness insights based on the problematic issues and trends, etc. Analert can be presented with, for example, the messages that triggeredthe alert, business metrics such as a volume of messages causing thealert or a cost for customer support agents to respond to the volume ofmessages, a graphical representation of textual data, an organizeddisplay of textual data, a word cloud

For example, example 400 of FIG. 4A illustrates graph 402 thatillustrates a graphical representation of trends of behavior determinedfrom messages between customers and customer support agents. Graph 402is similar to graph 302 of FIG. 3. In this example, the x-axisrepresents time and the y-axis represents a level of confidence. Thex-axis can be represented in a number of different time periods,including, for example, time periods of minutes, hours, days, weeks,months, years, etc. The y-axis can be represented as a number, e.g., anumber between zero and one, a number between zero and 100, apercentage, etc. Each number corresponds to a representative confidencescore that represents the likelihood that for a particular time intervalan event occurred.

Dots 404 represent spikes or trends of behavior for a particular timeperiod. The higher the score for a dot, the more likely an eventoccurred. As described herein, a trained model (e.g., logisticregression model) can be used to determine a score (e.g., representativeconfidence score) for each time period, and a trend detection techniquecan be utilized to determine whether a score for a current time periodis associated with an anomalous event. In the situation the score forthe current time period is associated with an anomalous event, an alertor other such notification can be triggered. In accordance with anembodiment, in the situation a dot for a time period corresponding to alow score is selected, the messages for that time period are typicallyunintelligible or meaningless. In the situation a dot for a time periodcorresponding to a high score is selected, the messages for that timeperiod typically correspond to an event or trend of behavior in thecustomer support messages.

For example, FIG. 4B illustrates example 420 of a listing of messages422 associated with dot 406. The messages include text that correspondsto an event and an issue associated with the event. In this example, theevent is a pay-per-view show and the issue is that the pay-per-view showis down. In certain embodiments, the words of the messages can beemphasized. For example, the size, color, font, etc. of the words can beadjusted based on a word's contribution to the confidence score,frequency of use, and the like. For example, each word in a message,once analyzed using a trained model, may be associated with a modelcoefficient. The model coefficient can be used to determine, forexample, the size, color, font, etc. of the words, as shown in FIG. 4Bwhere the size of the words, background color of the words, etc. isadjusted.

As shown in example 440 of FIG. 4C, a visual representation of the wordscan be displayed, such as word cloud 442. A “word cloud” or “tag cloud”is commonly used to refer to a visualization of text in a “cloud” oftext. A word cloud may display distinct words of a set of messages.Often, a word cloud gives greater prominence to important words. At aglance, a viewer of a word cloud is able to see what the “key words”(the most prominent) were in any particular set of messages. A workcloud can also be used to summarize the messages for a period of time.This provides a benefit to an appropriate person who would be able toglance at the word cloud and glean what words triggered an alert for aparticular time period. It also may be beneficial for the person toreview word clouds for past time periods to have a better understandingof behavior trends. In accordance with various embodiments, a word cloudcan quickly convey relative importance of words by displaying relativelyimportant words larger than relatively unimportant words. The locationand color can also be used to convey relative importance of words.

In various embodiments, in addition to, or instead of textualrepresentations, business metrics associated can be generated andpresented. The business metrics can be associated with sets of messages,such as the messages associated with a dot or other period of time,messages associated with confidence scores, representative confidencescores, etc. The business metrics or other relevant information can beany information beneficial or useful to a company, person, or entity.For example, business metrics can include the cost to respond tocustomer support request for a period of time, the number of messages totrigger an alert, the number of customer support agents used to respondto customer support requests for a period of time, the total amount oftime to respond to the customers support requests, etc.

As described herein, text used to train a model (e.g., prediction model)can be obtained from messages between customers and customer supportagents. FIG. 5 illustrates an example 500 of processing customer supportmessages to obtain the text, in accordance with various embodiments. Inthis example, text 502 from customer support messages between customersand customer support agents during customer support sessions can beobtained using text extraction techniques, speech recognitiontechniques, and the like. Text 502 can be associated with a timestampfor when the message for which the text was determined was received,information linking or otherwise mapping the text to the message,information identifying the source of the message, and the like.

In accordance with an embodiment, in example 500 of FIG. 5, audio-basedmessages 504, text-based messages 506, video-based messages 508, andcomputer-readable data 509 are received at communications manager 510.Communications manager 510 is operable to process the messages todetermine, for example, text and other information for each of themessages, such as a timestamp, a source, etc. In certain embodiments,communications manager 510 can separate or otherwise split messages intoone or more sets of messages. The sets can be based on one or morecriteria. For example, the sets can be based on when a message wasreceived. In an example, this can include a set of messages includingmessages received during a period of time, a set of messages includingmessages received before the period of time, a set of messages includingmessages received after the period of time, etc. Communications manager510 in various embodiments can flag or otherwise group messages to beincluded in a holdout set of messages, where messages in the holdout setare disjoint from messages in the other sets.

In an embodiment, an audio-based message includes messages with audiodata. Examples of audio-based messages include recorded telephone calls,a voice message containing audio of a person's voice, voice texts,voicemails, etc. A text-based message includes messages with text.Examples of text-based messages include text messages, emails, chatmessages, letters, signs, etc. A video-based message includes messageswith video data. Examples of video-based messages include recorded videocalls, video messages, etc. Computer-readable data can include data thatcan be processed by a computer. The data can be structured orunstructured. Computer-readable data can include data communicatedbetween electronic devices such as computer/server communication logs,computer instructions or code, backend logs, communication streambetween computers, etc.

Communications manager 510 can include message analyzer 512. Messageanalyzer 512 can be configured to analyze the messages to determine textfor each message, as well as determine or otherwise maintain informationabout the messages (e.g., timestamp, source, etc.) For example, messageanalyzer 512 or another appropriate component can use a text extractiontechnique to extract the text from text-based messages or a speechrecognition technique to convert audio-based and video-based messages totext. It should be noted that any one of a number of text extractiontechniques and speech recognition techniques can be utilized inaccordance with embodiments described herein.

Once the messages are obtained and the text determined, the text fromthe customer support messages can be used to train a model to predictwhether a message is received before a particular time period or duringthe time period. For example, example 600 of FIG. 6 illustrates anexample classification pipeline that can be utilized in accordance withvarious embodiments. In this example, messages are obtained that can beused to train model 606. Training data 602 and 609 can include customersupport session data (e.g., times and text of messages exchanged betweencustomers and customer support agents, etc.) or other communicationssession data between customers and customer support agents duringcustomer support sessions. As described, the messages can include calltranscripts, text conversation transcripts, timestamp records, etc.

In order to function as training data for one or more models (e.g., alogistic regression model or other machine learning-based model), atleast some of the customer support messages can include (or beassociated with) data that indicates a label, type, or classification ofthe customer support messages. The classifications in at least someembodiments can be selected from a set of classifications, orsub-classifications used to identify various types of messages, customersupport requests, timing information, etc. For example, for a period oftime, training data 602 can include messages received before the periodof time and training data 609 can include messages received during theperiod of time.

The training data can be labeled using one of any number of approaches.For example, the training data can be labeled using a feature extractorto extract features of the training data, such as timestamp, and acategory model to perform the classification of the training data basedon the extracted features. A feature extractor can extract, for example,the timestamp from each message. A category model can label each messagebased on the timestamp and the period of time. The category model may betrained, for example, on data from a company database that has beenannotated by human experts. The output of the category model can be, forexample, a one-of-k vector (where one element of the vector is true, or1 and the remaining values are false or 0) indicating which category thedata belongs to (of k possible categories). For example, the featureextractor can extract the timestamp for each message. The category modelcan process the timestamp and categorize the message as being receivedbefore the time period or during the time period. In another example,the timestamp for each message can be compared to the time period.Messages associated with a timestamp before the time period can belabeled one. Messages associated with a timestamp during the time periodcan be labeled zero.

In some embodiments, a portion of the training set will be retained astesting set 610 to use to test the accuracy of the trained model.

In this example, the training data is accessible to a training module604 which can feed the data to model 606 (e.g., a machine learning-basedmodel) to train the network. Model 606 can include, for example, one ofa logistic regression, Naïve Baye, random forest, neural network, orsupport vector machines (SVMs), etc.

The classification data can be provided to the model so the model canlearn features associated with the customer support messages. Thenetwork can then learn various combinations or relations of features ofthe customer support messages, such that when a message is processedwith the trained model, the model can recognize the features and outputthe appropriate confidence score for the message. In an embodiment, theconfidence score can quantify whether a message is received before thetime period or during the time period.

In certain embodiments, once at least the initial training hascompleted, a testing module 608 can utilize the testing set 610 to testthe trained model. Since the testing data already includesclassification data, such as data operable to label, identify, orotherwise indicate ground truth, the classifications and predictionsgenerated by the model can be compared against that data to determinethe accuracy of the model, both overall and for different categories ofcustomer support requests. In certain embodiments, the testing data canalso be used to train the model further. The results can be analyzed andif the results are acceptable, such as where the accuracy at least meetsa minimum accuracy threshold for some or all of the classifications andpredictions, the model can be provided to classifier 612 that is able toprocess customer support messages 614, and generate confidence score622.

In some embodiments, the training data 602 and 609 can be used astraining data for one of a logistic regression, Naïve Baye, randomforest, neural network, or support vector machines (SVMs), convolutionalrecurrent neural network, deep neural network, or other types of neuralnetworks or models, and/or combination of any of the above models,stacked models and heuristic rules. Various other approaches can be usedas well as discussed and suggested elsewhere herein.

FIG. 7 illustrates an example process 700 for determining training datathat can be utilized in accordance with various embodiments. It shouldbe understood that, for any process discussed herein, there can beadditional, fewer, or alternative steps, performed in similar ordifferent orders, or in parallel, within the scope of the variousembodiments unless otherwise stated. In this example, customer supportsession data (e.g., times and text of messages exchanged betweencustomers and customer support agents) can be obtained 702 for analysis.The session data can be obtained from, for example, company records,customer support, a stream of customer support messages, computer/servercommunication logs, computer-readable data between one or more computers(e.g., message logs, instructions, code, etc.), or otherwise obtained.In certain embodiments, the training data can be from one or more datastores maintained directly or indirectly by a customer support serviceprovider, resource provider, or a third-party, or from multiple sources,among other such options.

A timestamp associated with each message (or other informationassociated with each messages) can be used to determine 704 whether atype of classification of the messages corresponds to a category andincludes particular attributes, or types of attributes, for which amodel (e.g., logistic regression, neural network, or other machinelearning algorithm) can be trained. For example, using a timestamp foreach message, it can be determined whether a message occurred before,during, or after a selected period of time, and, e.g., whether thatmessage includes text, a certain amount of text, a type of text, etc.

If it is determined 706 that a message exhibits the attribute (e.g.,time) for a particular category (e.g., received before, during, after aperiod of time), then that message can be added 708 to the training set.In an embodiment, the training set can include different sets ofmessages. For example, the training set can include a set of messagesreceived before a time period and a set of messages received during thetime. If not, that message can be excluded 710 from the training set. Asmentioned elsewhere herein, in at least some embodiments, some of themessages may be instead added to a testing set, a holdout set, or notadded to any set but may have the attribute classification associatedtherewith. Messages in the holdout set can include messages receivedduring the time period and disjoint from the other messages in thetraining set.

If it is determined 712 that a full training set has been obtained,using any appropriate criterion as discussed or suggested herein, suchas a threshold number of messages for a period of time, then thetraining set generation can complete, and the messages can be stored 714for training and other purposes. Otherwise, the process can continueuntil a full set is obtained, all of the relevant session data isanalyzed, or another stop condition is satisfied.

FIG. 8 illustrates an example process 800 for training a predictionmodel that can be utilized in accordance with various embodiments. Oncethe training data is obtained 802, the training data can be provided asinput to a model training process. The training data can include, forexample, customer support session data (e.g., times and text of messagesexchanged between the customer and the customer support agent, etc.) orother communications session data between customers and customer supportagents and data that indicates a type or classification for a specificcategory for a customer support request, timestamp records, text ofmessages exchanged between customers and customer support agents, andthe like. As described, the type or classification for a specificcategory can include messages received before, during, or after aparticular time period. Thus, according to various embodiments, thetraining data can include a set of messages received before a timeperiod and a set of messages received during the time.

In the example of logistic regression, or other machine learning-basedmodel, the model can be trained 804 on the training data to predictwhether a message is received before a particular time period or duringthe time period. For example, the model can learn various combinationsor relations of features of message data such that when messages areprocessed with the trained model, the trained model can recognize thefeatures and output prediction information, including, for example, aprediction score or other such score, although various other outputs canbe utilized as well within the scope of the various embodiments.

If it is determined 805 that a stop condition has been met so thattraining should be completed, such as by processing the entire trainingset, then the trained model can be provided to process, for example,messages for a particular period of time. As discussed herein, the modelmight first go through a process to test 808 using at least some of thetraining classified with the attribute type from earlier steps of theprocess. If the training is not complete, then the training process cancontinue 809 until a trained model is obtained. Thereafter, the trainedmodel can be provided 810 to process messages associated with aparticular period of time to predict whether a message is receivedbefore a particular time period or during the time period.

FIG. 9 illustrates an example environment 900 in which aspects of thevarious embodiments can be implemented. In this example, users (e.g., acustomer of a company) can utilize an application or interface, such asa browser, executing on customer device 902 to customer support over atleast one network 906. The messages can be for support during a customersupport session, and the messages can be between a customer and acompany, or entity associated with the company.

The customer device 902 can be any appropriate computing device capableof requesting, receiving, and/or presenting content, such as may includea desktop computer, a thin client device or “dummy terminal,” asmartphone, an e-book reader, a tablet computer, a notebook computer, apersonal data assistant, a video gaming console or controller, a smarttelevision, a wearable computer (e.g., a smartwatch or glasses), or aportable media player, among others.

The messages can be communicated using any appropriate technique, suchas typing or speaking a message to an app running on a customer device(e.g., an app of the company or a third-party app created for processingcustomer support requests), typing or speaking a message on a web page,sending a text message, sending an email, etc. As used herein, a textmessage includes any message sent as text including but not limited to amessage sent using SMS (short message service) or a special-purposeapplication (e.g., Facebook messenger, Apple iMessage, Google Hangouts,or WhatsApp).

In this example, customer device 902 is in communication with a resourceprovider 907 via the at least one network 906. The at least one networkcan include a wired and/or wireless network, as may include theInternet, a cellular network, a local area network (LAN), an Ethernet,an Intranet, and the like. The customer device 902 may have varied localcomputing resources, such as may include central processing units andarchitectures, memory, mass storage, graphics processing units,communication network availability and bandwidth, etc.

Resource provider 907 can provide customer support services for variousservices. These services can include, for example, payment services,technical support services, human resource services, among other suchservices. In certain embodiments, provider 907 can be an intermediarybetween a customer of a company and a company, such as third-party 908.For example, third-party 908 can utilize resource provider 907 toprovide customer support to its customers to improve the customersupport experience and to reduce overall customer support costs. Forexample, third-party 908 may find it more cost effective to use loadmanagement services of resource provider 907 than to implement its ownload management services. In an example, third-party 908 can utilizeprovider 907 to provide customer support services to any number ofcompanies as well as customers of those companies. This can include, forexample, automatically responding to customer support requests,connecting a customer with a customer service representative (CSR),customer support agent, etc., selecting a CSR to process a customersupport request, establishing customer support sessions betweencustomers and CSRs, providing resources to CSRs to process customersupport requests, etc. In another example, third-party 908 may utilizethe services of a fourth-party company in providing support services totheir customers, companies, and customers of those companies. In yetanother example, third-party 908 can utilize provider 907 to detectanomalous events and trends of behavior.

Messages and information associated with the messages can be received toan interface and/or networking layer 910 of the resource provider 907.The interface and/or networking layer can include any of a number ofcomponents known or used for such purposes, as may include one or morerouters, switches, load balancers, Web servers, application programminginterfaces (APIs), and the like.

The messages and associated information can be stored in communicationsdata store 922 and as well as information associated with the customerin in profile data store 916 until such messages is ready to beprocessed. The information may include any relevant information, such astext of a message from the customer, information about a customer (e.g.,an IP address, a customer ID, or authentication credentials), orinformation about a company from which the customer is seeking customersupport (e.g., a company ID). The information may be received directlyfrom a customer device or may be received via a server of a company fromwhom the customer is requesting support. In some embodiments, someinformation may be received directly from the customer device and someinformation may be received from a server of the company. In thesituation where such information is not available, the system canrequest the information or other information including, for example,information about the customer.

The message and information associated with the message can be directedto communications manager 918 or other appropriate component.Communications manager 918 is operable to process messages to determine,for example, text and other information for each of the messages, suchas a timestamp, a source, etc. In certain embodiments, communicationsmanager 918 can separate or otherwise split messages into one or moresets of messages. The sets can be based on filtering criteria. Forexample, the sets can be based on when a message was received. In anexample, this can include a set of messages including messages receivedduring a period of time, a set of messages including messages receivedbefore the period of time, a set of messages including messages receivedafter the period of time, etc. Communications manager 918 in variousembodiments can flag messages to be included in a holdout set, wheremessages in the holdout set are disjoint from messages in the othersets. The messages and associated information can be stored incommunications data store 922 or other appropriate data store.

Once the messages are obtained and the text determined, the text fromthe customer support messages can be used to train a model to predictwhether a message is received before a particular time period or duringthe time period, for example, training component 932, model 934, andclassifiers data store 930. For example, training component 932 canobtain messages from communication data store 922 to be used as trainingdata. The messages can be organized into sets. In an example, this caninclude a set of messages including messages received during a period oftime, a set of messages including messages received before the period oftime, a set of messages including messages received after the period oftime, etc.

For each one of a plurality of periods of time, a model (e.g., aprediction model) can be trained. Each model can be trained on messagesthat correspond to each models' period of time, and the trained modelscan be stored in classifier data store 930.

Once a model for a period of time is trained, the model can be used toevaluate each message in a holdout set of messages for the period oftime to obtain a set of confidence scores corresponding to the holdoutset of messages. The set of confidence scores can be processed todetermine a representative confidence score for the time period based onthe set of confidence scores. The representative confidence score can bestored in confidence score data store 936 with representative confidencescores from a plurality of prediction models trained over a plurality ofprevious time periods.

For a selected time period, trend detection component 938 can apply amodel (e.g., trend detection model) to the representative confidencescores for periods of time before the selected time period to determinewhether the representative confidence score for the selected time periodis associated with an anomalous event.

In the situation an anomalous event is detected, notification component940 can generate and present an alert for the anomalous event to anappropriate entity. Visualization component 942 can present businessmetrics associated with the alert, a graphical representation of textualdata, an organized display of textual data, a word cloud, a list ofmessages, and the like.

In accordance with various embodiments, additional or alternativecomponents and elements can be used in such a system in accordance withthe various embodiments. Accordingly, it should be noted that additionalservices, providers, and/or components can be included in such a system,and although some of the services, providers, components, etc., areillustrated as being separate entities and/or components, theillustrated arrangement is provided as an example arrangement and otherarrangements as known to one skilled in the art are contemplated by theembodiments described herein. In accordance with various embodiments,the various components described herein may be performed by any numberof server computing devices, desktop computing devices, mainframecomputers, and the like. Individual devices may implement one of thecomponents of the system. In some embodiments, the system can includeseveral devices physically or logically grouped to implement one of themodules or components of the system. For example, the system can includevarious modules and components combined on a single device, multipleinstances of a single module or component, etc. In one specific,non-limiting embodiment, communications manager 918, trend detectioncomponent 938, and visualization component 942, can execute on onedevice and training component 932 can execute on another device, andnotification component 940 can execute on yet another device. In anotherembodiment, the components can execute on the same device or othercombination of devices.

In some embodiments, the features and services provided by the systemmay be implemented as web services consumable via a communicationnetwork. In further embodiments, the system is provided by one morevirtual machines implemented in a hosted computing environment. Thehosted computing environment may include one or more rapidly provisionedand released computing resources, which computing resources may includecomputing, networking and/or storage devices. A hosted computingenvironment may also be referred to as a cloud computing environment.

FIG. 10 illustrates an example process 1000 for detecting anomalousevents in accordance with various embodiments. In this example, aplurality of messages is obtained 1002. The messages can be obtainedfrom, for example, company records, customer support logs, a stream ofcustomer support messages, computer/server communication logs,computer-readable data between computers (e.g., message logs,instructions, code, etc.), or otherwise obtained. The messages can beseparated 1004 or otherwise split into sets of messages. For example, afirst set of messages, a second set of messages, and a holdout set ofmessages, wherein the first set of messages includes messages receivedbefore a time period, the second set of messages includes messagesreceived during the time period, and the holdout set of messagesincludes messages received during the time period and disjoint from thesecond set of messages.

A prediction model 1006 (e.g., a logistic regression model) is trainedon the first set of messages and the second set of messages to predictwhether a message is received before a time period or during the timeperiod. For example, the prediction model can be used to evaluate 1008each message in the holdout set of messages to obtain a set ofconfidence scores corresponding to the holdout set of messages. The setof confidence scores can be processed to determine 1010 a representativeconfidence score for the time period based on the set of confidencescores. A set of previously determined representative confidence scoresfrom a plurality of prediction models trained over a plurality ofprevious time periods can be obtained 1012.

A trend detection model can be applied 1014 to the set of representativeconfidence scores to determine 1016 whether the representativeconfidence score for the time period is associated with an anomalousevent. In the situation where the representative confidence score is notassociated with an anomalous event, the process can continue 1017 forthe next period of time. In the situation where the representativeconfidence score is associated with an anomalous event, an alert for theanomalous event can be generated 1018 and presented to an appropriateentity. The alert can be presented with, for example, the messages thattriggered the alert, business metrics such as a volume of messagescausing the alert or a cost for customer support agents to respond tothe volume of messages, a graphical representation of textual data, anorganized display of textual data, a word cloud, among other suchinformation.

FIG. 11 illustrates an example process 1100 for detecting anomalousevents in accordance with various embodiments. In this example,confidence scores for messages in a holdout set of messages isdetermined 1102. This can include, for example, evaluating a predictionmodel on each message in the holdout set of messages. The confidencescores can be used 1104 to order the holdout set of messages. Forexample, the messages can be ordered from messages having a highestconfidence score to messages having a lowest confidence score. Arepresentative confidence score for a period of time associated with theholdout set of messages can be determined 1106 using the order of theholdout set of messages. For example, the representative confidencescore for the time period can be the confidence score corresponding to aconfidence score threshold. In an example, the representative confidencescore can be the confidence score corresponding to the 95_(th)percentile confidence score.

In certain embodiments, the representative confidence score may be anaverage of representative confidence scores. For example, a plurality ofmessages associated with a period of time can be split into a pluralityof groups of first sets of messages, second sets of messages, andholdout sets of messages. A representative confidence score can bedetermined for each holdout set of messages of the plurality of groupsto obtain a set of representative confidence scores, and therepresentative confidence score for the period of time can correspond toone of the average of the scores or the median representative confidencescore of the set of representative confidence scores. In yet anotherexample, representative confidence scores from the set of representativeconfidence scores that meet a threshold representative confidence scorecan be selected, and the average of the selected representativeconfidence scores can correspond to the representative confidence scorefor the period of time.

FIG. 12 illustrates an example process 1200 for detecting anomalousevents in accordance with various embodiments. In this example, aplurality of messages is obtained 1202. The plurality of messages can beassociated with a plurality of time periods. If it is determined 1204that the number of messages for a current time period has not beenreceived, using any appropriate criterion as discussed or suggestedherein, such as a threshold number of messages, then messages from aprevious period of time can be obtained 1206 or otherwise used orbuffered with messages of the current time period. The number ofmessages obtained from the previous time period can be equal or morethan then a difference between the threshold number of messages and thenumber of messages for the current time period. Otherwise, a model(e.g., a logistic regression model) can be trained 1208 on a first andsecond set of messages for the current time period. The first set ofmessages can include messages received before the current time periodand the second set of messages can include messages received during thecurrent time period. The model can be evaluated 1210 on a holdout set ofmessages associated with the current time period to obtain a set ofconfidence scores corresponding to the holdout set of messages. Arepresentative confidence score for the current time period can bedetermined 1212 using the set of confidence scores in accordance withembodiments described herein. A determination can be made 1214 whether anew model is to be trained for an available time period. If it isdetermined that there are no remaining time periods, then the processcan complete, and the representative confidence scores can be stored1216 for trend detection and other purposes. In certain embodiments, themodel for each time period can be stored for use in one or more otherpurposes. Otherwise, if it is determined that a time period remains, theprocess can continue 1218 until a model is trained and a representativeconfidence score determined for the remaining time periods or untilanother stop condition is satisfied.

FIG. 13 illustrates a set of basic components of an electronic computingdevice 1300, such as computing device 104 in FIG. 1A. In variousembodiments, computer device 1300 may be used to implement any of thesystems, devices, or methods described herein. In some embodiments,computer device 1300 may correspond to any of the various devicesdescribed herein, including, but not limited, to mobile devices, tabletcomputing devices, wearable devices, personal or laptop computers,vehicle-based computing devices, or other devices or systems describedherein. As shown in FIG. 13, computer device 1300 can include varioussubsystems connected by a bus 1302. The subsystems may include an I/Odevice subsystem 1304, a display device subsystem 1306, and a storagesubsystem 1310 including one or more computer-readable storage media1308. The subsystems may also include a memory subsystem 1312, acommunication subsystem 1320, and a processing subsystem 1322.

In device 1300, bus 1302 facilitates communication between the varioussubsystems. Although a single bus 1302 is shown, alternative busconfigurations may also be used. Bus 1302 may include any bus or othercomponents to facilitate such communication as is known to one ofordinary skill in the art. Examples of such bus systems may include alocal bus, parallel bus, serial bus, bus network, and/or multiple bussystems coordinated by a bus controller. Bus 1302 may include one ormore buses implementing various standards such as Parallel ATA, serialATA, Industry Standard Architecture (ISA) bus, Extended ISA (EISA) bus,MicroChannel Architecture (MCA) bus, Peripheral Component Interconnect(PCI) bus, or any other architecture or standard as is known in the art.

In some embodiments, I/O device subsystem 1304 may include various inputand/or output devices or interfaces for communicating with such devices.Such devices may include, without limitation, a touch screen or othertouch-sensitive input device, a keyboard, a mouse, a trackball, a motionsensor or other movement-based gesture recognition device, a scrollwheel, a click wheel, a dial, a button, a switch, audio recognitiondevices configured to receive voice commands, microphones, image capturebased devices such as eye activity monitors configured to recognizecommands based on eye movement or blinking, and other types of inputdevices. I/O device subsystem 1304 may also include identification orauthentication devices, such as fingerprint scanners, voiceprintscanners, iris scanners, or other biometric sensors or detectors. Invarious embodiments, I/O device subsystem may include audio outputdevices, such as speakers, media players, or other output devices.

Computer device 1300 may include a display device subsystem 1306.Display device subsystem may include one or more lights, such as one ormore light emitting diodes (LEDs), LED arrays, a liquid crystal display(LCD) or plasma display or other flat-screen display, a touch screen, ahead-mounted display or other wearable display device, a projectiondevice, a cathode ray tube (CRT), and any other display technologyconfigured to visually convey information. In various embodiments,display device subsystem 1306 may include a controller and/or interfacefor controlling and/or communicating with an external display, such asany of the above-mentioned display technologies.

As shown in FIG. 13, device 1300 may include storage subsystem 1310including various computer-readable storage media 1308, such as harddisk drives, solid-state drives (including RAM-based and/or flash-basedSSDs), or other storage devices. In various embodiments,computer-readable storage media 1308 can be configured to storesoftware, including programs, code, or other instructions, that isexecutable by a processor to provide the functionality described herein.For example, the instructions, when executed, can enable a computingdevice to detect and generate alerts for anomalous events in accordancewith the present disclosure may be embodied on a computer-readablemedium. This may include separating the plurality of messages into afirst set of messages, a second set of messages, and a holdout set ofmessages, wherein the first set of messages includes messages receivedbefore a time period, the second set of messages includes messagesreceived during the time period, and the holdout set of messagesincludes messages received during the time period and disjoint from thesecond set of messages; training a prediction model on the first set ofmessages and the second set of messages to predict whether a message isreceived before the time period or during the time period; evaluatingthe prediction model on each message in the holdout set of messages toobtain a set of confidence scores corresponding to the holdout set ofmessages; determining a representative confidence score for the timeperiod based on the set of confidence scores; obtaining a plurality ofprevious representative confidence scores from a plurality of predictionmodels trained over a plurality of previous time periods; applying amodel to the plurality of previous representative confidence scores todetermine that the representative confidence score for the time periodis associated with an anomalous event; generating an alert for theanomalous event; using the set of confidence scores corresponding to theholdout set of messages to select a set of messages; and presenting theset of messages.

In some embodiments, storage system 1310 may include various data storesor repositories or interface with various data stores or repositoriesthat store data used with embodiments described herein. Such data storesmay include, databases, object storage systems and services, data lakesor other data warehouse services or systems, distributed data stores,cloud-based storage systems and services, file systems, and any otherdata storage system or service. In some embodiments, storage system 1310can include a media reader, card reader, or other storage interfaces tocommunicate with one or more external and/or removable storage devices.In various embodiments, computer-readable storage media 1308 can includeany appropriate storage medium or combination of storage media. Forexample, computer-readable storage media 1308 can include, but is notlimited to, any one or more of random access memory (RAM), read-onlymemory (ROM), electronically erasable programmable ROM (EEPROM), flashmemory or other memory technology, optical storage (e.g., CD-ROM,digital versatile disk (DVD), Blu-ray® disk or other optical storagedevice), magnetic storage (e.g., tape drives, cassettes, magnetic diskstorage or other magnetic storage devices). In some embodiments,computer-readable storage media can include data signals or any othermedium through which data can be transmitted and/or received.

Memory subsystem 1312 can include various types of memory, includingRAM, ROM, flash memory, or other memory. Memory 1312 can include SRAM(static RAM) or DRAM (dynamic RAM). In some embodiments, memory 1312 caninclude a BIOS (basic input/output system) or other firmware configuredto manage initialization of various components during, e.g., startup. Asshown in FIG. 13, memory 1312 can include applications 1314 andapplication data 1316. Applications 1314 may include programs, code, orother instructions, that can be executed by a processor. Applications1314 can include various applications such as browser clients, campaignmanagement applications, data management applications, and any otherapplication. Application data 1316 can include any data produced and/orconsumed by applications 1314. Memory 1312 can additionally includeoperating system 1318, such as macOS®, Windows®, Linux®, various UNIX®or UNIX- or Linux-based operating systems, or other operating systems.

Device 1300 can also include a communication subsystem 1320 configuredto facilitate communication between device 1300 and various externalcomputer systems and/or networks (such as the Internet, a local areanetwork (LAN), a wide area network (WAN), a mobile network, or any othernetwork). Communication subsystem 1320 can include hardware and/orsoftware to enable communication over various wired (such as Ethernet orother wired communication technology) or wireless communicationchannels, such as radio transceivers to facilitate communication overwireless networks, mobile or cellular voice and/or data networks, WiFinetworks, or other wireless communication networks. Additionally, oralternatively, communication subsystem 1320 can include hardware and/orsoftware components to communicate with satellite-based or ground-basedlocation services, such as GPS (global positioning system). In someembodiments, communication subsystem 1320 may include, or interfacewith, various hardware or software sensors. The sensors may beconfigured to provide continuous or and/or periodic data or data streamsto a computer system through communication subsystem 1320

As shown in FIG. 13, processing system 1322 can include one or moreprocessors or other devices operable to control computing device 1300.Such processors can include single-core processors 1324, multi-coreprocessors, which can include central processing units (CPUs), graphicalprocessing units (GPUs), application specific integrated circuits(ASICs), digital signal processors (DSPs) or any other generalized orspecialized microprocessor or integrated circuit. Various processorswithin processing system 1322, such as processors 1324 and 1326, may beused independently or in combination depending on the application.

Various other configurations are may also be used, with particularelements that are depicted as being implemented in hardware may insteadbe implemented in software, firmware, or a combination thereof. One ofordinary skill in the art will recognize various alternatives to thespecific embodiments described herein.

Various other configurations are may also be used, with particularelements that are depicted as being implemented in hardware may insteadbe implemented in software, firmware, or a combination thereof. One ofordinary skill in the art will recognize various alternatives to thespecific embodiments described herein.

The various embodiments can be implemented in a wide variety ofoperating environments, which in some cases can include one or more usercomputers or computing devices which can be used to operate any of anumber of applications. User or client devices can include any of anumber of general-purpose personal computers, such as desktop or laptopcomputers running a standard operating system, as well as cellular,wireless and handheld devices running mobile software and capable ofsupporting a number of networking and messaging protocols. Such a systemcan also include a number of workstations running any of a variety ofcommercially available operating systems and other known applicationsfor purposes such as development and database management. These devicescan also include other electronic devices, such as dummy terminals,thin-clients, gaming systems and other devices capable of communicatingvia a network.

Most embodiments utilize at least one network that would be familiar tothose skilled in the art for supporting communications using any of avariety of commercially available protocols, such as TCP/IP, FTP, UPnP,NFS, and CIFS. The network can be, for example, a local area network, awide-area network, a virtual private network, the internet, an intranet,an extranet, a public switched telephone network, an infrared network, awireless network and any combination thereof.

In embodiments utilizing a web server, the web server can run any of avariety of server or mid-tier applications, including HTTP servers, FTPservers, CGI servers, data servers, Java servers and businessapplication servers. The server(s) may also be capable of executingprograms or scripts in response requests from user devices, such as byexecuting one or more web applications that may be implemented as one ormore scripts or programs written in any programming language, such asJava®, C, C # or C++ or any scripting language, such as Perl, Python orTCL, as well as combinations thereof. The server(s) may also includedatabase servers, including without limitation those commerciallyavailable from Oracle®, Microsoft®, Sybase® and IBM®.

The environment can include a variety of data stores and other memoryand storage media as discussed above. These can reside in a variety oflocations, such as on a storage medium local to (and/or resident in) oneor more of the computers or remote from any or all of the computersacross the network. In a particular set of embodiments, the informationmay reside in a storage-area network (SAN) familiar to those skilled inthe art. Similarly, any necessary files for performing the functionsattributed to the computers, servers or other network devices may bestored locally and/or remotely, as appropriate. Where a system includescomputerized devices, each such device can include hardware elementsthat may be electrically coupled via a bus, the elements including, forexample, at least one central processing unit (CPU), at least one inputdevice (e.g., a mouse, keyboard, controller, touch-sensitive displayelement or keypad) and at least one output device (e.g., a displaydevice, printer or speaker). Such a system may also include one or morestorage devices, such as disk drives, optical storage devices andsolid-state storage devices such as random-access memory (RAM) orread-only memory (ROM), as well as removable media devices, memorycards, flash cards, etc.

Such devices can also include a computer-readable storage media reader,a communications device (e.g., a modem, a network card (wireless orwired), an infrared communication device) and working memory asdescribed above. The computer-readable storage media reader can beconnected with, or configured to receive, a computer-readable storagemedium representing remote, local, fixed and/or removable storagedevices as well as storage media for temporarily and/or more permanentlycontaining, storing, transmitting and retrieving computer-readableinformation. The system and various devices also typically will includea number of software applications, modules, services or other elementslocated within at least one working memory device, including anoperating system and application programs such as a client applicationor web browser. It should be appreciated that alternate embodiments mayhave numerous variations from that described above. For example,customized hardware might also be used, and/or particular elements mightbe implemented in hardware, software (including portable software, suchas applets) or both. Further, connection to other computing devices suchas network input/output devices may be employed.

Storage media and other non-transitory computer-readable media forcontaining code, or portions of code, can include any appropriate mediaknown or used in the art, including storage media and communicationmedia, such as but not limited to volatile and non-volatile, removableand non-removable media implemented in any method or technology forstorage of information such as computer-readable instructions, datastructures, program modules or other data, including RAM, ROM, EEPROM,flash memory or other memory technology, CD-ROM, digital versatile disk(DVD) or other optical storage, magnetic cassettes, magnetic tape,magnetic disk storage or other magnetic storage devices or any othermedium which can be used to store the desired information and which canbe accessed by a system device. Based on the disclosure and teachingsprovided herein, a person of ordinary skill in the art will appreciateother ways and/or methods to implement the various embodiments.

The methods, systems, and devices discussed above are examples. Variousconfigurations may omit, substitute, or add various procedures orcomponents as appropriate. For instance, in alternative configurations,the methods may be performed in an order different from that described,and that various steps may be added, omitted, or combined. Also,features described with respect to certain configurations may becombined in various other configurations. Different aspects and elementsof the configurations may be combined in a similar manner. Also,technology evolves and, thus, many of the elements are examples and donot limit the scope of the disclosure or claims.

The methods, systems, and devices discussed above are described withreference to block diagrams and/or operational illustrations of methods,systems, and computer program products according to embodiments of thepresent disclosure. The functions/acts noted in the blocks may occur outof the order as shown in any flowchart. For example, two blocks shown insuccession may in fact be executed substantially concurrent or theblocks may sometimes be executed in the reverse order, depending uponthe functionality/acts involved. Additionally, or alternatively, not allof the blocks shown in any flowchart need to be performed and/orexecuted. For example, if a given flowchart has five blocks containingfunctions/acts, it may be the case that only three of the five blocksare performed and/or executed. In this example, any of the three of thefive blocks may be performed and/or executed.

Specific details are given in the description to provide a thoroughunderstanding of example configurations (including implementations).However, configurations may be practiced without these specific details.For example, well-known circuits, processes, algorithms, structures, andtechniques have been shown without unnecessary detail to avoid obscuringthe configurations. This description provides example configurationsonly, and does not limit the scope, applicability, or configurations ofthe claims. Rather, the above description of the configurations willprovide those skilled in the art with an enabling description forimplementing described techniques. Various changes may be made in thefunction and arrangement of elements without departing from the spiritor scope of the disclosure.

Having described several example configurations, various modifications,alternative constructions, and equivalents may be used without departingfrom the spirit of the disclosure. For example, the above elements maybe components of a larger system, wherein other rules may takeprecedence over or otherwise modify the application of variousimplementations or techniques of the present disclosure. Also, a numberof steps may be undertaken before, during, or after the above elementsare considered.

Reference in the specification to “one embodiment” or to “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiments is included in at least one exampleimplementation or technique in accordance with the present disclosure.The appearances of the phrase “in one embodiment” in various places inthe specification are not necessarily all referring to the sameembodiment.

Unless specifically stated otherwise as apparent from the followingdiscussion, it is appreciated that throughout the description,discussions utilizing terms such as “processing” or “computing” or“calculating” or “determining” or “displaying” or the like, refer to theaction and processes of a computer system, or similar electroniccomputing device, that manipulates and transforms data represented asphysical (electronic) quantities within the computer system memories orregisters or other such information storage, transmission or displaydevices. Portions of the present disclosure include processes andinstructions that may be embodied in software, firmware or hardware, andwhen embodied in software, may be downloaded to reside on and beoperated from different platforms used by a variety of operatingsystems.

In addition, the language used in the specification has been principallyselected for readability and instructional purposes and may not havebeen selected to delineate or circumscribe the disclosed subject matter.Accordingly, the present disclosure is intended to be illustrative, andnot limiting, of the scope of the concepts discussed herein.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the disclosed embodiments (especially in thecontext of the following claims) are to be construed to cover both thesingular and the plural, unless otherwise indicated herein or clearlycontradicted by context. The terms “comprising,” “having,” “including,”and “containing” are to be construed as open-ended terms (i.e., meaning“including, but not limited to,”) unless otherwise noted. The term“connected” is to be construed as partly or wholly contained within,attached to, or joined together, even if there is something intervening.Recitation of ranges of values herein are merely intended to serve as ashorthand method of referring individually to each separate valuefalling within the range, unless otherwise indicated herein and eachseparate value is incorporated into the specification as if it wereindividually recited herein. All methods described herein can beperformed in any suitable order unless otherwise indicated herein orotherwise clearly contradicted by context. The use of any and allexamples, or exemplary language (e.g., “such as”) provided herein, isintended merely to better illuminate embodiments of the disclosure anddoes not pose a limitation on the scope of the disclosure unlessotherwise claimed. No language in the specification should be construedas indicating any non-claimed element as essential to the practice ofthe disclosure.

Disjunctive language such as the phrase “at least one of X, Y, or Z,”unless specifically stated otherwise, is intended to be understoodwithin the context as used in general to present that an item, term,etc., may be either X, Y, or Z, or any combination thereof (e.g., X, Y,and/or Z). Thus, such disjunctive language is not generally intended to,and should not, imply that certain embodiments require at least one ofX, at least one of Y, or at least one of Z to each be present.

Having been provided with the description and illustration of thepresent application, one skilled in the art may envision variations,modifications, and alternate embodiments falling within the generalinventive concept discussed in this application that do not depart fromthe scope of the following claims.

What is claimed is:
 1. A computer-implemented method, comprising:obtaining a plurality of messages; separating the plurality of messagesinto a first set of messages, a second set of messages, and a holdoutset of messages, wherein the first set of messages includes messagesreceived before a time period, the second set of messages includesmessages received during the time period, and the holdout set ofmessages includes messages received during the time period and disjointfrom the second set of messages; training a prediction model on thefirst set of messages and the second set of messages to predict whethera message is received before the time period or during the time period;evaluating the prediction model on each message in the holdout set ofmessages to obtain a set of confidence scores corresponding to theholdout set of messages; determining a representative confidence scorefor the time period based on the set of confidence scores; obtaining aplurality of previous representative confidence scores from a pluralityof prediction models trained over a plurality of previous time periods;applying a model to the plurality of previous representative confidencescores to determine that the representative confidence score for thetime period is associated with an anomalous event; generating an alertfor the anomalous event; using the set of confidence scorescorresponding to the holdout set of messages to select a set ofmessages; and presenting the set of messages.
 2. Thecomputer-implemented method of claim 1, further comprising: using theset of confidence scores to order the holdout set of messages; anddetermining the representative confidence score using the order of theholdout set of messages.
 3. The computer-implemented method of claim 1,further comprising: splitting the plurality of messages into a pluralityof groups of first sets of messages, second sets of messages, andholdout sets of messages; determining a representative confidence scorefor each holdout set of messages of the plurality of groups to obtain aset of representative confidence scores; and averaging the set ofrepresentative confidence scores to determine the representativeconfidence score.
 4. The computer-implemented method of claim 1, whereinthe model is a change point detection model.
 5. The computer-implementedmethod of claim 1, wherein the prediction model is one of a logisticregression, Naïve Baye, random forest, neural network, or support vectormachines (SVMs).
 6. The computer-implemented method of claim 1, whereinpresenting the set of messages further includes: presenting businessmetrics associated with the holdout set of messages, wherein thebusiness metrics include one of a volume of messages causing the alertor a cost for customer support agents to respond to the volume ofmessages.
 7. The computer-implemented method of claim 1, whereinpresenting the set of messages further includes: presenting one of agraphical representation of textual data, an organized display oftextual data, a word cloud, or a list of messages.
 8. Thecomputer-implemented method of claim 1, further comprising: determiningtext for individual messages of the plurality of messages using one of atext extraction technique or an automated speech recognition technique.9. The computer-implemented method of claim 1, further comprising: foreach previous time period of the plurality of previous time periods,training a prediction model for a previous time period on messagesassociated with the previous time period; evaluating the predictionmodel on each message in a holdout set of messages for the previous timeperiod to obtain a representative confidence score for the previous timeperiod; and storing the representative confidence score for the previoustime period.
 10. A system, comprising: at least one computer comprisingat least one processor and at least one memory, the at least onecomputer configured to: obtain a plurality of messages; separate theplurality of messages into a first set of messages, a second set ofmessages, and a holdout set of messages, wherein the first set ofmessages includes messages received before a time period, the second setof messages includes messages received during the time period, and theholdout set of messages includes messages received during the timeperiod and disjoint from the second set of messages; train a predictionmodel on the first set of messages and the second set of messages topredict whether a message is received before the time period or duringthe time period; evaluate the prediction model on each message in theholdout set of messages to obtain a set of confidence scorescorresponding to the holdout set of messages; determine a representativeconfidence score for the time period based on the set of confidencescores; obtain a plurality of previous representative confidence scoresfrom a plurality of prediction models trained over a plurality ofprevious time periods; apply a model to the plurality of previousrepresentative confidence scores to determine that the representativeconfidence score for the time period is associated with an anomalousevent; generate an alert for the anomalous event; use the set ofconfidence scores corresponding to the holdout set of messages to selecta set of messages; and present the set of messages.
 11. The system ofclaim 10, wherein the at least one computer is further configured to:use the set of confidence scores to order the holdout set of messages;and determine the representative confidence score using the order of theholdout set of messages.
 12. The system of claim 10, wherein the atleast one computer is further configured to: split the plurality ofmessages into a plurality of groups of first sets of messages, secondsets of messages, and holdout sets of messages; determine arepresentative confidence score for each holdout set of messages of theplurality of groups to obtain a set of representative confidence scores;and average the set of representative confidence scores to determine therepresentative confidence score.
 13. The system of claim 10, wherein theat least one computer is further configured to present one of agraphical representation of textual data, an organized display oftextual data, a word cloud, a list of messages, or business metricsassociated with the holdout set of messages, wherein the businessmetrics include one of a volume of messages causing the alert or a costfor customer support agents to respond to the volume of messages. 14.The system of claim 10, wherein the at least one computer is furtherconfigured to: determine text for individual messages of the pluralityof messages using one of a text extraction technique or an automatedspeech recognition technique.
 15. The system of claim 10, wherein the atleast one computer is further configured to: determine that a totalnumber of messages received within a current time period is below athreshold number of messages; and use messages from a previous timeperiod with the current time period when training the prediction model.16. The system of claim 10, wherein the time period is for one of aminute, an hour, a day, or a week.
 17. One or more non-transitory,computer-readable media comprising computer executable instructionsthat, when executed, cause at least one processor to perform actionscomprising: obtaining a plurality of messages; separating the pluralityof messages into a first set of messages, a second set of messages, anda holdout set of messages, wherein the first set of messages includesmessages received before a time period, the second set of messagesincludes messages received during the time period, and the holdout setof messages includes messages received during the time period anddisjoint from the second set of messages; training a prediction model onthe first set of messages and the second set of messages to predictwhether a message is received before the time period or during the timeperiod; evaluating the prediction model on each message in the holdoutset of messages to obtain a set of confidence scores corresponding tothe holdout set of messages; determining a representative confidencescore for the time period based on the set of confidence scores;obtaining a plurality of previous representative confidence scores froma plurality of prediction models trained over a plurality of previoustime periods; applying a model to the plurality of previousrepresentative confidence scores to determine that the representativeconfidence score for the time period is associated with an anomalousevent; generating an alert for the anomalous event; using the set ofconfidence scores corresponding to the holdout set of messages to selecta set of messages; and presenting the set of messages.
 18. The one ormore non-transitory, computer-readable media of claim 17, wherein the atleast one processor further performs actions comprising: using the setof confidence scores to order the holdout set of messages; anddetermining the representative confidence score using the order of theholdout set of messages.
 19. The one or more non-transitory,computer-readable media of claim 17, wherein the at least one processorfurther performs actions comprising: splitting the plurality of messagesinto a plurality of groups of first sets of messages, second sets ofmessages, and holdout sets of messages; determining a representativeconfidence score for each holdout set of messages of the plurality ofgroups to obtain a set of representative confidence scores; andaveraging the set of representative confidence scores to determine therepresentative confidence score.
 20. The one or more non-transitory,computer-readable media of claim 17, wherein the at least one processorfurther performs actions comprising: determining that a total number ofmessages received within a current time period is below a thresholdnumber of messages; and using messages from a previous time period withthe current time period.